SYSTEM AND METHOD FOR USING TELEPHONY CONTROLS ON A 
PERSONAL COMPUTER 

CROSS-REFERENCE TO RELATED APPLICATIONS 

5 The present invention is related to the following 

copending United States Patent Applications filed May 5, 2003, 
assigned to the assignee of the present invention, and hereby 
incorporated by reference in their entireties: 

"System and method for Auxiliary Display of Information 
10 for a Computing Device, " Attorney Docket No. 3810/ 

"Real-Time Communications Architecture and Methods for 
use with a Personal Computer System, " Attorney Docket No. 
3820; 

"Record Button on a Computer System," Attorney Docket No. 

15 3830; 

"System and method for Auxiliary Processing Of 
Information for a Computing Device,'' Attorney Docket No. 3840; 

"System and Method for Activating a Computer System; " 
Attorney Docket No. 3850; 
20 "Computer Camera System and Method for Reducing 

Parallax," Attorney Docket No. 3870; 

"Control and Communications Panel for a Computer System," 
Attorney Docket No. 3880; and 

"Notification Lights, Locations and Rules for a Computer 
25 System," Attorney Docket No. 3900. 
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FIELD OF THE INVENTION 

The invention relates generally to computer systems and 
communications, and more particularly to an improved system 
5 and method for using telephony controls on a personal computer 
system. 

BACKGROUND OF THE INVENTION 

The role of contemporary computers continues to evolve as 
10 a tool used for communications and media applications. 
Personal computers offer many new and exciting media 
applications for personal entertainment such as live 
television, personal video recording, playing or recording 
digital music, displaying or creating digital photographs, and 
15 playing movies recorded on a DVD. With the introduction of 
Voice-over-IP (VoIP) and other IP telephony applications, 
computers also may be used for handling phone calls and voice 
mail, in addition to handling media applications and customary 
communication applications such as email, instant messaging, 
20 fax communications, and so forth. 

A personal computer with integrated telephony can enhance 
voice and video communication in ways that empower the user 
while increasing user productivity and enjoyment. For 
example, a personal computer may be designed that is capable 
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of accepting incoming calls of various kinds such as analog 
Public Switched Telephone Network (PSTN), private branch 
exchange (PBX) , cellular telephony, VoIP, and/or cellular 
telephony. Application programs may use VoIP telephony, 
5 instant messaging and videoconferencing by integrating the use 
of microphone, speakers and camera. 

What is needed is a simplified system and method for a 
user to interact with these various communications and media 
applications, in a consistent way across these communication 

10 modalities. The system and method should operate in what is 
perceived to be an intelligent manner, and be fairly 
comprehensive in handling communications, as well as flexible 
and extensible to handle many user scenarios and usage 
patterns. At the same time, such a system and method should 

15 be straightforward for users to activate and also interact 
with in an intuitive way. 

SUMMARY OF THE INVENTION 

Briefly, the present invention provides a system and 
20 method including telephony controls for handling telephony 

applications on a personal computer with integrated telephony 
capabilities. Such a personal computer may be connected to 
telephone equipment for accepting incoming calls of various 
kinds such as analog Public Switched Telephone Network (PSTN) , 


private branch exchange (PBX) , cellular telephony or VoIP. To 
this end, the computer is generally arranged as a central 
communications point for multiple communications-related 
devices. Accordingly, the present invention provides an 
5 enhanced user interface with telephony controls for such a 
computer so that users may activate and interact with 
telecommunications applications such as handling phone calls 
and voice mail. 

The telephony controls may include a hook switch, a talk 

10 button, a speakerphone button, a mute button, a flash button, 
a conference button, a microphone button, and a voice mail 
button. There may also be other telephony control buttons, 
such as an optional record button in close proximity to the 
telephony controls for use in responding to voice mail. 

15 Because the state of an individual telephony control button 

may not be readily apparent to a user, an indicator light such 
as an LED may be located adjacent to or integrated into each 
telecommunication control button as appropriate. 
Additionally, there may be a single hook indicator light 

20 alternatively used. 

The hook switch operates in a number of ways, including 
using it to terminate a call, lifting it to make a receiver 
active and pick up waiting line, as a flash button to switch 
lines or terminate a call if no lines are waiting. The hook 


state may be toggled between "off hook" and "on hook" either 
by lifting the handset from the cradle with a mechanically 
integrated hook switch, by pressing the talk button, by 
pressing the speakerphone button, or in another way such as 
5 via graphical user interface. 

The system and method allow a number of user interaction 
scenarios with respect to real-time communications, including 
handling an incoming telephone call and making a telephone 
call. To make a telephone call, the user lifts the handset or 

10 presses the talk button or presses the speakerphone button, 

and then dials a number. Alternatively, the user may select a 
contact using a graphical user interface or employ voice 
dialing. While on a call, a user may switch the active 
receiver to a speakerphone or other receiver such as a handset 

15 or headset. The user may decide to switch the system audio 
input device from the handset or headset to a computer 
microphone. A user may also perform tasks such as switching 
lines using the flash button to answer a waiting call or 
toggling a call hold state by actuating the hold button. The 

20 user can also perform other communications-related tasks, 
including conference calling and checking voice mail. 
Whenever a new voice mail message arrives, an indicator light 
may blink to notify the user, whereby the user can press the 
voice mail button to access voice mail. 


In general, the system and method of the present 
invention is relatively comprehensive as well as flexible and 
extensible to handle virtually any telephony functionality, 
and additional telephony controls may be easily added as 
5 needed. The system and method may also be customizable to an 
extent. Other advantages will become apparent from the 
following detailed description when taken in conjunction with 
the drawings, in which: 

10 BRIEF DESCRIPTION OF THE DRAWINGS 

FIGURE 1 is a block diagram generally representing a 
computer system into which the present invention may be 
incorporated; 

FIG. 2 is a general representation of a computer system 
15 arranged with integrated communications-related devices and 
telephony controls, in accordance with an aspect of the 
present invention; 

FIG. 3 is a block diagram generally representing the 
central control of communication devices and connections to 
20 communications channels, in accordance with an aspect of the 
present invention; 

FIG. 4 is a block diagram generally representing 
components to couple various devices and control and display 
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connection state in accordance with an aspect of the present 
invention; 

FIG. 5 is a block diagram generally representing the 
components in an exemplary embodiment of the telephony 
5 controls, in accordance with an aspect of the present 
invention; 

FIG. 6 is a general representation of a partial keyboard 
with telephony controls incorporated into the keyboard, in 
accordance with an aspect of the present invention; 

10 FIG. 7 is a representation of a control panel positioned 

beneath a computer monitor with telephony controls 
incorporated into the control panel, in accordance with an 
aspect of the present invention; 

FIG. 8 is a representation of telephony controls 

15 incorporated into a monitor, in accordance with an aspect of 
the present invention; 

FIG. 9 is a representation of an alternative control 
panel with telephony controls, in accordance with an aspect of 
the present invention; and 

20 FIG. 10 is a general state diagram representing states of 

operation when making a call, receiving a call, and when a 
call is waiting, in accordance with an aspect of the present 
invention. 
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DETAILED DESCRIPTION 

EXEMPLARY OPERATING ENVIRONMENT 

FIGURE 1 illustrates an example of a suitable computing 
system environment 100 on which the invention may be 
5 implemented. The computing system environment 100 is only one 
example of a suitable computing environment and is not 
intended to suggest any limitation as to the scope of use or 
functionality of the invention. Neither should the computing 
environment 100 be interpreted as having any dependency or 

10 requirement relating to any one or combination of components 
illustrated in the exemplary operating environment 100. 

The invention is operational with numerous other general 
purpose or special purpose computing system environments or 
configurations. Examples of well known computing systems, 

15 environments, and/or configurations that may be suitable for 
use with the invention include, but are not limited to: 
personal computers, server computers, hand-held or laptop 
devices, tablet devices, multiprocessor systems, 
microprocessor-based systems, set top boxes, programmable 

20 consumer electronics, network PCs, minicomputers, mainframe 

computers, distributed computing environments that include any 
of the above systems or devices, and the like. 

The invention may be described in the general context of 
computer-executable instructions, such as program modules, 


being executed by a computer. Generally, program modules 
include routines, programs, objects, components, data 
structures, and so forth, which perform particular tasks or 
implement particular abstract data types. The invention may 
5 also be practiced in distributed computing environments where 
tasks are performed by remote processing devices that are 
linked through a communications network. In a distributed 
computing environment, program modules may be located in local 
and/or remote computer storage media including memory storage 
10 devices. 

With reference to FIG. 1, an exemplary system for 
implementing the invention includes a general purpose 
computing device in the form of a computer 110. Components of 
the computer 110 may include, but are not limited to, a 

15 processing unit 120, a system memory 130, and a system bus 121 
that couples various system components including the system 
memory to the processing unit 120. The system bus 121 may be 
any of several types of bus structures including a memory bus 
or memory controller, a peripheral bus, and a local bus using 

20 any of a variety of bus architectures. By way of example, and 
not limitation, such architectures include Industry Standard 
Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, 
Enhanced ISA (EISA) bus, Video Electronics Standards 
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Association (VESA) local bus, and Peripheral Component 
Interconnect (PCI) bus also known as Mezzanine bus. 

The computer 110 typically includes a variety of 
computer-readable media. Computer-readable media can be any 
5 available media that can be accessed by the computer 110 and 
includes both volatile and nonvolatile media, and removable 
and non-removable media. By way of example, and not 
limitation, computer-readable media may comprise computer 
storage media and communication media. Computer storage media 

10 includes volatile and nonvolatile, removable and non-removable 
media implemented in any method or technology for storage of 
information such as computer-readable instructions, data 
structures, program modules or other data. Computer storage 
media includes, but is not limited to, RAM, ROM, EE PROM, flash 

15 memory or other memory technology, CD-ROM, digital versatile 
disks (DVD) or other optical disk storage, magnetic cassettes, 
magnetic tape, magnetic disk storage or other magnetic storage 
devices, or any other medium which can be used to store the 
desired information and which can accessed by the computer 

20 110. Communication media typically embodies computer-readable 
instructions, data structures, program modules or other data 
in a modulated data signal such as a carrier wave or other 
transport mechanism and includes any information delivery 
media. The term "modulated data signal" means a signal that 
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has one or more of its characteristics set or changed in such 
a manner as to encode information in the signal. By way of 
example, and not limitation, communication media includes 
wired media such as a wired network or direct-wired 
5 connection, and wireless media such as acoustic, RF, infrared 
and other wireless media. Combinations of the any of the 
above should also be included within the scope of computer- 
readable media. 

The system memory 130 includes computer storage media in 

10 the form of volatile and/or nonvolatile memory such as read 
only memory (ROM) 131 and random access memory (RAM) 132. A 
basic input/output system 133 (BIOS), containing the basic 
routines that help to transfer information between elements 
within computer 110, such as during start-up, is typically 

15 stored in ROM 131. RAM 132 typically contains data and/or 
program modules that are immediately accessible to and/or 
presently being operated on by processing unit 120. By way of 
example, and not limitation, FIG. 1 illustrates operating 
system 134, application programs 135, other program modules 

20 136 and program data 137. 

The computer 110 may also include other removable/non- 
removable, volatile/nonvolatile computer storage media. By 
way of example only, FIG. 1 illustrates a hard disk drive 141 
that reads from or writes to non-removable, nonvolatile 

- 11 - 


magnetic media, a magnetic disk drive 151 that reads from or 
writes to a removable, nonvolatile magnetic disk 152, and an 
optical disk drive 155 that reads from or writes to a 
removable, nonvolatile optical disk 156 such as a CD ROM or 
5 other optical media. Other removable/non-removable, 

volatile/nonvolatile computer storage media that can be used 
in the exemplary operating environment include, but are not 
limited to, magnetic tape cassettes, flash memory cards, 
digital versatile disks, digital video tape, solid state RAM, 

10 solid state ROM, and the like. The hard disk drive 141 is 
typically connected to the system bus 121 through a non- 
removable memory interface such as interface 140, and magnetic 
disk drive 151 and optical disk drive 155 are typically 
connected to the system bus 121 by a removable memory 

15 interface, such as interface 150. 

The drives and their associated computer storage media, 
discussed above and illustrated in FIG. 1, provide storage of 
computer-readable instructions, data structures, program 
modules and other data for the computer 110. In FIG. 1, for 

20 example, hard disk drive 141 is illustrated as storing 

operating system 144, application programs 145, other program 
modules 14 6 and program data 147, Note that these components 
can either be the same as or different from operating system 
134, application programs 135, other program modules 136, and 
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program data 137. Operating system 144, application programs 
145, other program modules 146, and program data 147 are given 
different numbers herein to illustrate that, at a minimum, 
they are different copies. A user may enter commands and 
5 information into the computer 110 through input devices such 
as a tablet, or electronic digitizer, . 164, a microphone 163, a 
keyboard 162 and pointing device 161, commonly referred to as 
mouse, trackball or touch pad. Other input devices not shown 
in FIG. 1 may include a joystick, game pad, satellite dish, 

10 scanner, or the like. These and other input devices are often 
connected to the processing unit 120 through a user input 
interface 160 that is coupled to the system bus, but may be 
connected by other interface and bus structures, such as a 
parallel port, game port or a universal serial bus (USB) . A 

15 monitor 191 or other type of display device is also connected 
to the system bus 121 via an interface, such as a video 
interface 190. The monitor 191 may also be integrated with a 
touch-screen panel or the like. Note that the monitor and/or 
touch screen panel can be physically coupled to a housing in 

20 which the computing device 110 is incorporated, such as in a 
tablet-type personal computer. In addition, computers such as 
the computing device 110 may also include other peripheral 
output devices such as speakers 195 and printer 196, which may 
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be connected through an output peripheral interface 194 or the 
like. 

The computer 110 may operate in a networked environment 
using logical connections to one or more remote computers, 
5 such as a remote computer 180. The remote computer 180 may be 
a personal computer, a server, a router, a network PC, a peer 
device or other common network node, and typically includes 
many or all of the elements described above relative to the 
computer 110, although only a memory storage device 181 has 

10 been illustrated in FIG. 1. The logical connections depicted 
in FIG. 1 include a local area network (LAN) 171 and a wide 
area network (WAN) 173, but may also include other networks. 
Such networking environments are commonplace in offices, 
enterprise-wide computer networks, intranets and the Internet. 

15 When used in a LAN networking environment, the computer 110 is 
connected to the LAN 171 through a network interface or 
adapter 170. When used in a WAN networking environment, the 
computer 110 typically includes a modem 172 or other means for 
establishing communications over the WAN 173, such as the 

20 Internet. The modem 172, which may be internal or external, 
may be connected to the system bus 121 via the user input 
interface 160 or other appropriate mechanism. . In a networked 
environment, program modules depicted relative to the computer 
110, or portions thereof, may be stored in the remote memory 
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storage device. By way of example, and not limitation, FIG. 1 
illustrates remote application programs 185 as residing on 
memory device 181. It will be appreciated that the network 
connections shown are exemplary and other means of 
5 establishing a communications link between the computers may 
be used. 

Telephony Controls 

The present invention is generally directed towards 

10 providing a user interface including telephony controls for a 
personal computer with integrated telephony capabilities. 
Such a personal computer may be connected to telephone 
equipment for accepting incoming calls of various kinds such 
as analog Public Switched Telephone Network (PSTN) , private 

15 branch exchange (PBX) , cellular telephony or VoIP. To this 
end, the computer is generally arranged as a central 
communications point for multiple communications-related 
devices, such as generally described in the related copending 
U.S. Patent application Serial No. 10/429,905 entitled "Real- 

20 Time Communications Architecture and Methods for use with a 

Personal Computer System. " Accordingly, the present invention 
provides an enhanced user interface with telephony controls 
for such a computer so that users may activate and interact 
with telecommunications applications such as handling phone 
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calls and voice mail. As will be understood, the various 
telecommunications interfaces, telephony controls, 
communication channels, devices and scenarios described herein 
are only examples, and there are many others to which the 
5 present invention will apply. 

Turning to FIG. 2 of the drawings, there is shown a 
computer system, such as the computer system 110 of FIG. 1, in 
which various communications-related devices are integrated. 
For example, the computer system 110 includes a monitor 191 

10 having an attached camera 202, an attached telephone (e.g., 

handset) 204, and an attached speaker 194, which along with a 
microphone 163, may function as a speakerphone . Other devices 
not separately represented in FIG. 2 include mobile 
communications devices such as a cellular telephone or other 

15 mobile computing device capable of connecting in some way 
(e.g., by wired cradle or Bluetooth™ technology) with the 
computer system. Although shown as individual devices in FIG. 
2, it is understood that any or all of these devices, such as 
the speaker 194 and microphone 163, may be combined into a 

20 unitary assembly, or built into other devices, such as the 
monitor 191. In any event, each of these devices may be 
present and if so are connected directly or indirectly by a 
wired or wireless connection to interfaces at the main PC 
housing 214 (containing at least the motherboard, but also 
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typically containing some non-volatile storage) . As used 
herein, the term "connect" and its variants and "couple" are 
equivalent and generally mean that there is some type of 
information capable of being transferred between one entity 
5 and another, regardless of whether directly or indirectly via 
any intermediaries and/or any transformation of the 
information. 

Also represented in FIG. 2 is an auxiliary display device 
220, such as for displaying caller ID data or like information 

10 as described below. The auxiliary display 220 is typically 
small (relative to the main display screen 191), and may be 
the display of another device, such as the screen of a mobile 
computing device, but may also be a dedicated display, such as 
one that is operably coupled to the computer operating system 

15 when the computer system 110 is in a powered-up state. 

Further, note that the auxiliary display need not be an actual 
display, but can be a projection (e.g., onto a wall) of the 
information. The auxiliary display device 220 may provide 
some of its own processing and storage resources, so as to 

20 remain functional to some extent when the computer operating 
system is not running. The keyboard 162 and/or monitor 191 
(and/or possibly the pointing device 161) may also include 
non-conventional buttons 222, such as buttons related to 
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telephony controls, and non-conventional LED indicators 224, 
such as indicators related to telephony controls. 

As generally represented in FIG. 3, the computer system 
110 may include a real-time communications controller 300, 
5 primarily implemented in a software component or set of 
software components, that intelligently handles various 
aspects of a user's real-time communication needs, including 
facilitating communications in general, switching 
communications to and from appropriate devices, and/or 

10 transitioning communications among the various communications 
channels and/or types of communication. Note that the real- 
time communications controller 300 need not be built into the 
computer system 110, but can connect thereto as an add-on 
device, such as via a USB connection. 

15 To handle a user's various communications needs, the 

real-time communications controller 300 allows a user to 
connect to various communication channels (e.g., a telephone 
company switching network, a mobile phone network, the 
internet or an intranet, and virtually any other local area or 

20 wide area network over which communication is possible) using 
the various communications-related data input/output devices. 
In general, the user is able to input data via any input 
mechanism, e.g., any microphone on any device, via any text or 
pointing-based user data entry mechanism 302 such as the 
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keyboard and/or mouse, and/or the camera 202, and have that 
data configured as appropriate (e.g., converted from speech to 
text) and sent to an appropriate (one or possibly more) of the 
available communication channels. Similarly, the real-time 
5 communications controller 300 is able to receive data and 

output appropriate data to the user on any one or more of the 
output mechanisms, such as the speaker of a speakerphone 304 
when the user has selected that device, the handset telephone 
204 when lifted, a headset 306 if activated, the mobile phone 

10 or similar computing device 308 if selected, a PBX phone 310 
connected to a PBX mechanism 312 and/or a network card 314. 
When video, graphical and/or text information is available, 
data can also be displayed on the main display 191 (if 
active), and/or the auxiliary display 220. Note that some of 

15 the devices are represented in FIG. 3 as directly connected to 
the computer system while other are shown as being wireless; 
however it is understood that any device may be wired or 
wireless. For example, a typical mobile computing device or 
mobile telephone may be cradled and thus "wired" to the 

20 computer, and/or may integrate a cellular phone, Wireless 

Ethernet (also known as 802.11b or Wi-Fi), and/or Bluetooth™ 
wireless technology. 

In accordance with an aspect of the present invention, 
the real-time communications controller 300 controls various 
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communication modes for the user, thereby removing much of the 
user's need to deal with device and connection considerations. 
As generally represented in FIG. 4, this is accomplished by 
responding to the user's actions, via mode control logic 400 
5 in the real-time communications controller 300 that follows 
various real-time communication rules 402 and/or is based at 
least in part on other user-configurable settings and a 
current operating state 404. The mode control logic 400 
likewise responds to external events, such as incoming calls, 
10 pages, text messages, forwarded voicemail messages, and so 
forth. 

User actions can occur and be detected in any number of 
ways, including essentially any way of providing user action 
data or intended action data to a computer system. For 

15 example, such actions can include lifting or replacing the 

attached handset, which correspondingly actuates a hook switch 
408 that is connected to the real-time communications 
controller 300, and thus can be detected as state data. 
Telephony controls 410, such as in the form of buttons on a 

20 device such as a talk button or buttons on the keyboard or 

monitor, (e.g., a speakerphone button, a flash button, and so 
on) can similarly provide user action information, as can a 
PBX telephone 310 or other computing device / telephone 308 
(FIG. 3) connected to the computer system. Controls 420 
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accompanying the auxiliary display 220, such as in the form of 
scroll buttons and possibly an enter or select button (or 
equivalent button elsewhere on the system such as the flash 
button) , can also provide user action data, as can the 
5 computer data entry mechanisms 302 such as a keyboard and 
mouse. Any microphone, such as the microphone 163, can also 
provide user action data, e.g., via speech recognition 
(command and control) performed with a speech to text 
recognizer 430. Text may also be converted to speech by an 

10 appropriate converter 432, such as when the user enters text 
to leave a message, but instead actually leaves a voice 
message on a called device, e.g., when the logic 400 detects 
that forwarding text would not be appropriate. User action 
information may also be communicated by wireless radio 434, 

15 and/or from a network connection or other communication 

channel. Motion and/or proximity sensing can likewise serve 
as a user action sensor. 

Where appropriate, a graphical user interface component 
440 shown as part of the mode control logic 400 (but possibly 

20 anywhere in the computer system, such as in another 

application program) may facilitate entry of the user action 
information. For example, the user may interact with one of 
the displays to select a displayed telephone number for 
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connecting or reconnecting with a person or device at that 
number. 

Similarly the system may output information to the user 
in any feasible manner, including via the user interface when 
5 appropriate. Essentially, anything capable of outputting 

something that can be sensed by a user can serve as an output 
mechanism, including the displays 191 and/or 220, lights and 
other indicators 224, the speaker 194 or other audio 
generating device such as a ringer on a cell phone, and other 

10 mechanisms such as one that provides tactile feedback (e.g., a 
vibrating cell phone) . Thus, as used herein, the term 
''display" is only an example, and is not limited to visible 
information, but can include any sensed output. For example, 
a hook indicator in the form of an LED may be configured to 

15 flash red to indicate an incoming call, be lit green when off 
hook, blink amber when the call is on hold, and blink on-and- 
off when a call is muted. A tone and pattern can "display" 
the same information audibly. 

Moreover, the present invention allows for various levels 

20 of computing skill. In general, the telephony controls 410 
and the hook switch 408 that are provided allow the user to 
take advantage of much of the real-time communications 
controller's functionality without ever having to operate the 
graphical user interface software 440. However, to accomplish 
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more sophisticated operations as desired, a user can also 
connect via the user interface 440, e.g., to automatically 
dial calls, answer via a mouse click, customize settings and 
preferences and so forth. 
5 In general, the logic 400 receives user action data or 

remotely communicated data (such as an incoming call), and 
based on various real-time communications rules 402 and user 
configurable settings 404, responds in a proper manner. For 
example, as described below, for an incoming call, the mode 

10 control logic 400 will display the caller ID information on 

the auxiliary display 220, or possibly the main display 191 if 
so configured by the user, sound a ring or call waiting tone 
on the speaker 194, and wait for a user action. If the user 
presses a telephony control button such as a speakerphone 

15 button, the call will be connected to the appropriate device, 
e.g., routed to the speaker 194 and microphone 163. Any other 
audio, such as music that the user was listening to, may be 
automatically muted until the call is complete, e.g., until 
the user again presses the speakerphone button to disconnect. 

20 The real-time communications controller 300 thus 

automatically and transparently responds in an appropriate 
manner to a user's needs. By way of example, consider the 
example above of a user who is working at the computer when a 
call comes in. Instead of figuring out what type of call it 
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is, finding the device necessary to (possibly) answer it, and 
so on, the user merely glances at the auxiliary display, 
decides whether to take the call, and if so, selects any 
appropriate device to answer. Because the real-time 
5 communications controller 300 is the center of communication, 
a call to any of the user's landline telephone devices or any 
(smart) cellular telephone will relay the caller ID data to 
the real-time communications controller 300, which will 
display that caller ID data on the auxiliary display 220 as 

10 any other call. For the same reason, the user can answer any 
incoming call on the speakerphone 304, the handset 204, the 
headset 306, the cellular telephone or mobile computing device 
308 and so on. The user need not even know whether a given 
call came in on a given communications channel, such as the 

15 cellular channel instead of a POTS channel or network channel, 
although for reasons such as cost of airtime, the user may 
want to know this information. Regardless of the device and 
the channel, the user is provided access to a universal 
address book, call log, and inbox. 

20 Moreover, the real-time communications controller allows 

a user to transition between communication channels. Thus, 
for example, a user who receives a POTS telephone call can 
answer, talk for awhile, and switch that call over to a mobile 
phone to continue the conversation away from the computer 
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system. Note that this is not conventional call forwarding 
set up in advance for switching at the central switching 
office or the like, but rather on-demand transitioning during 
a telephone call. For example, rules 402 can be set up to 
5 favor landline over cellular for local calls since there is no 
airtime charge, however cellular long distance may be favored 
over landline when there is no additional charge and the user 
is likely to have unused airtime before the next billing 
cycle. Voice over IP may be the most favored, but may not be 

10 available when the computer system is powered down. 

One simple way to accomplish transitioning live 
connections is via a different telephone line relative to the 
source of the call. For example, if a user is leaving his or 
her office, the real-time communications controller 300 can 

15 transparently (or mostly transparently) establish a cellular 
connection to the user's mobile telephone device on that other 
line and connect the other caller to that mobile telephone 
device through the real-time communications controller 300. 
Voice over IP can similarly be used. 

20 Three-way and conference calling capabilities can also be 

leveraged, e.g., a user can walk into his or her office while 
connected to another person via a cell phone, the real-time 
communications controller 400 can establish another call to 
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the other party via a POTS, PBX or voice over IP call/ and 
when connected, terminate the cellular phone call. 

In addition to switching and transitioning among devices 
and channels, the real-time communications controller 300 can 
5 switch and/or transition among types of communication. For 
example, a user can respond to a telephone call with an 
instant message, either automatically or by typing / speech 
recognition, e.g., to notify the caller that the call was 
received but could not be answered at that moment. In the 

10 event that the real-time communications controller 300 has the 
caller ID and/or other information from that call, possibly 
including preset knowledge such as that the incoming telephone 
number belongs to a mobile device, the real-time 
communications controller 300 can adjust the instant message 

15 as appropriate, e.g., convert a text message to speech to 
leave a voice message when the call was known to be from a 
mobile phone that does not handle text, or when the user is 
known to be traveling. Note that the instant message does not 
have to be sent to the same device that called the user, e.g., 

20 a user can respond to a received phone call from a 

conventional telephone with an instant message to the caller's 
computer system. A simple database lookup or the like by the 
mode control logic can provide the recipient's address from 
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the caller ID data so that the user does not even have to 
enter this information. 

As another type change, when available, the present 
invention can start with an audio call and add video (one-way 
5 or two-way) and/or instant messaging as. desired by the 

parties, or start with an instant message and add audio and/or 
video. For example, a user can press a camera button, or 
select video via a mouse click, to establish a video 
connection and start transmitting video. A video call can 

10 similarly be transitioned to a non-video (e.g., audio-only) 

call. In general, the present invention facilitates switching 
and/or transitioning between any modes of communication. 

Turning to an explanation of the operation of the present 
invention with particular reference to the telephony controls 

15 410, FIG. 5 presents the components in an exemplary embodiment 
of the telephony controls 410. Those skilled in the art will 

appreciate that the functionality implemented within the 

t 

blocks illustrated in the diagram may be implemented as 
separate components or the functionality of several or all of 
20 the blocks may be implemented within a single component. For 
example, a separate library may be implemented that contains 
the functionality of the control handling logic and the 
functionality of the display logic. In such an 
implementation, executables linked to the library may send 
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control events directly to the control interface when they are 
executing on the computer system. 

FIG. 5 shows how the telephony controls 410 may be used 
to perform communications-related tasks. In general, state 
5 changes or the like in the controls are provided to the 
control handling program (e.g., the operating system 134, 
real-time communications controller 300, or an application 
program 135), along with data that identifies the control. 
Via controls handling logic 502, the control handling program 

10 then acts in the current context corresponding to the control. 
This may include generating one or more hardware or software 
events or commands, such as answering an incoming call, 
switching from a call on one line to a call on another line, 
conferencing calling, and so forth. 

15 The control handling program also controls any indicators 

508 that are present as described above, via indicator display 
logic 504 to convey information to the user. For example, the 
control handling logic 502 and the indicator/display logic 504 
communicate with the hardware interface 506 to receive input 

20 from the telephony controls 410 and send output to the 

indicators 508 for display. Note that the indicators need not 
be LEDs, but include any type of mechanism that outputs 
information that is capable of being sensed by a user, 
including visual, audible and tactile output. Each of the 
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indicators may be related to one of the controls, and thus 
each indicator may be incorporated into or positioned 
proximate its corresponding control. Where the indicators are 
LEDs, colors and illumination patterns (frequencies, duty 
5 cycles, number of pulses, and so forth) may be varied to 

convey different information to users regarding each control. 

Further, although an operating system 134 (or real-time 
.communications controller 300 or some application program 135) 
is shown in FIG. 5 as handling the telephony controls, it . 

10 should be noted that the computer system need not be fully 
operational for the telephony controls 410 and/or the 
indicators 508 to work in accordance with the present 
invention. Indeed, some of the controls may still work when 
the computer is powered down, at least to a default extent or 

15 to an extent configured by a user. For example, the user may 
want the telephony controls to work as a conventional 
appliance when the computer system is powered down, so that 
the user may answer incoming calls using the telephone 
handset, speakerphone and caller-ID device. Further, the user 

20 may limit the extent of the communications based on the 

computer system state, e.g., when the user is not logged in, 
long distance calls may be blocked, calls to a certain prefix 
may be blocked, and so forth. Any tasks that do not require 
a running operating system or application program similarly 
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may remain operational and use the telephony controls when the 
operating system or application program is not available or in 
standby mode. 

To handle the controls and communications in such a 
5 powered-down mode, the control handling logic may be loaded 
into executable non-volatile memory, operated with a secondary 
processor, and so forth, so that communication works as long 
as some power is available, even though the disk, main 
processor, main display, network card and/or other parts of 

10 the system are powered down. 

FIG. 6 presents a general representation of a partial 
keyboard with telephony controls 410 incorporated into the 
keyboard 162. The telephony controls 410 may include a Talk 
button 602, a Speaker or Speakerphone button 604, a Mute 

15 button 606, a Hold button 608, a Flash button 610, a 
Conference button 612, a Microphone button 614, and a 
Voicemail button 616. As is appreciated, other telephony 
control buttons may be included such as a Transfer button for 
transferring a call to another person, or a Send button for 

20 initiating a call, or an End button for ending a call, and so 
forth. In one embodiment, there may also be an optional 
Record button (not shown) in close proximity to the telephony 
controls 410 for use in responding to voice mail. 
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Where there is an attached telephone or handset 204 as 
shown in FIG. 2, a cradle hook switch is mechanically 
integrated into the handset cradle. A cradle hook switch is a 
two-position toggle switch that closes when the handset is off 
5 hook and that is open when the handset is in the cradle. 

Lifting the handset from the cradle usually takes the handset 
off hook and displays a dialog box for selecting the person to 
call. When there is an incoming call, lifting the handset 
will answer the call, taking the handset off hook. If there 

10 is another incoming phone call while a call is already in 
progress, depressing the cradle hook switch quickly will 
toggle between the two calls. During a call, replacing the 
handset in the cradle will hang up the call. If the handset 
is not the active receiver during a call, lifting the handset 

15 from the cradle will make it the active receiver and switches 
off the old active receiver such as the speakerphone 304 or 
headset 306. 

The telephony controls 410 shown in FIG. 6 are used to 
handle .telephony communications in general, including making 
20 calls, answering calls, switching communications to and from 
appropriate lines or devices, operating voicemail and/or other 
types of telephony applications. When there is an incoming 
call, pressing the Talk button 602 will answer the call, 
taking the receiver off hook. The Talk button 602 is usually 


integrated into the keypad of telephone devices, such as a 
headset 306 or handset 204. The Talk button 602 performs 
similar functionality to the hook switch and can generally be 
considered similar to the hook switch operation. In general, 
5 either a talk button or cradle hook switch (or at least a 
graphical user interface equivalent) is required for each 
phone handset 204 or headset 306. The Talk button 206 
comprises a single-throw, momentary switch that toggles the 
hook state for a telephone receiver. Like the cradle hook 

10 switch, its primary function is to initiate or answer a call, 
and then hang up when the conversation is finished. During a 
call, pressing the Talk button 602 on the active receiver will 
hang up the call. Pressing the Talk button 206 on another 
receiver will have the effect of making it the active 

15 receiver, switching off the old active receiver (handset 204 
or headset 306) . When there is an incoming call, pressing the 
Talk button 602 will answer the call, taking the receiver off 
hook. If there is another incoming call while a call is 
already in progress, pressing the Talk button 602 will toggle 

20 between the two calls. 

The Speaker or Speakerphone button 604 is a special case 
of the Talk button 602, used where the incoming call is 
broadcast over a speaker and audio is collected via a far 
field microphone. In a cordless handset with speaker phone 
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capabilities, it may be important to indicate to the user that 
the speakerphone is active to differentiate it from the 
handset talk switch. The Speakerphone button 604 similarly 
comprises a single-throw, momentary switch to toggle the hook 
5 state for a speakerphone. Its primary function is to initiate 
or answer a call, and then hang up when the conversation is 
finished. 

If a virtual speakerphone is implemented using the 
microphone 163 and speakers 194 that are already part of the 

10 computer system, the Speakerphone button 604 may be located on 
the keyboard as shown in FIG. 6 (although it is alternatively 
feasible to have such a keyboard Speakerphone button 604 work 
with another speakerphone) . This button usually takes the 
speakerphone off hook and displays a dialog box for selecting 

15 the person to call. During a call, pressing the Speakerphone 
button 604 when the speakerphone is the active receiver will 
hang up the call. If the speakerphone is not the active 
receiver, this will have the effect of making it the active 
receiver, switching off the old active receiver. When there 

20 is an incoming call, pressing the Speakerphone button 604 will 
answer the call, taking the speakerphone off hook. If there 
is another incoming call while a call is already in progress, 
pressing the Speakerphone button 604 will toggle between the 
two calls. 
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Because the talk button or speakerphone button state may 
not be readily apparent to a user, a hook indicator light such 
as an LED may be used in conjunction with a Talk or 
Speakerphone button, and, for example, may be located adjacent 
5 or integrated into the Talk button or Speakerphone button. 
The Talk button 602 and the Speakerphone button 604 are each 
shown in FIG. 6 with a hook indicator light located above 
them. A single hook indicator light may alternatively be 
used. The hook indicator may indicate state via various 

10 colors and flash patterns, e.g., steady state unlit when the 
receiver is on hook, steady state green when the receiver is 
off hook, or steady state red when no service is available. A 
flashing indicator, such as with a "wink" duty cycle can 
overlay the steady state, e.g., a red wink overlay can 

15 indicate call waiting, or an amber flash overlay can indicate 
a call holding. The call waiting overlay indicator will 
continue to blink as long as the user can pick up the phone to 
connect to the incoming caller, even if the call is no longer 
ringing. This allows call screening when callers are 

20 recording voice mail. The call holding overlay is generally 
only needed when a dedicated hold indicator is not present. 

The Mute button 606, also known as a Call Mute button, 
allows the user to quickly mute the default system audio input 
device such as microphone 163 or the microphone in the handset 
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204 or headset 306. The Mute button 606 is operable as a 
momentary switch that toggles the mute state for a telephone 
receiver between mute and not-mute. The Mute button 606 can 
be a stand-alone button as illustrated in FIG. 6, or 
5 alternatively, may be integrated into the handset 204 or the 
headset 306. The indicator light should be located adjacent 
to the Mute button 606 or integrated into the control itself. 
Pressing the Mute button 606 mutes audio input and sets the 
call mute indicator to amber. Pressing it again restores the 

10 audio input and restores the call mute indicator to green. 

The Hold button 608 places current call on hold and frees 
the receiver for performing other telephony operations such' as 
making another call, or hanging up to pick up on another 
receiver, and so forth. The Hold button 608 is operable as a 

15 momentary switch that toggles the hold state for a telephone 
receiver between an on-hold state and a not-on-hold state. An 
indicator light may be located adjacent to the Hold button 608 
or integrated into the control itself. Pressing the Hold 
button 608 places the current call into a call waiting state 

20 and sets the hold button indicator from unlit to amber. 

Pressing the Hold button 608 again moves the call from the 
call waiting state to the current call on the active receiver 
and resets the hold indicator as unlit. 
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The Flash button 610 is often used for alternate line 
selection and generates a. momentary on hook condition to 
signal the application. If there is an incoming call, an 
indicator light for the Flash button may blink red to notify 
5 the user of a waiting call. When the user is notified of a 
call waiting, pressing the Flash button 610 will place the 
current call on hold and answer the incoming call. With calls 
on hold, pressing the Flash button 610 places the current call 
on hold and picks up the next call in the hold queue. When 

10 there are no incoming calls waiting, the Flash button 
indicator light is reset as unlit. 

The Conference button 612 initiates a conference call. 
Pressing the Conference button 612 joins the current call with 
one or more calls waiting on hold. When the Conference button 

15 612 is pressed, the indicator light associated with it is set 
to green. The indicator light is reset as unlit when the last 
caller or the receiver hangs ups. 

The Microphone button 614 usually allows the user to 
switch the default system audio input device from the handset 

20 204 or headset 306 to personal computer microphone 163. The 
Microphone button 614 may be operable as a momentary switch 
that toggles the microphone state for a telephone receiver 
between the default audio input device and a secondary audio 
input device. The indicator light should be located adjacent 


to the Microphone button 614 or integrated into the control 
itself* Pressing the Microphone button 614 activates audio 
input from the personal computer microphone 163 and sets the 
Microphone button indicator to green. Pressing it again 
5 restores the audio input to the default system audio input 
device and resets the Microphone button indicator as unlit. 

The Voicemail button 616 is used to access received 
voicemail. Whenever a new voicemail is received, the 
indicator light is set to green to alert the user. Pressing 

10 the Voicemail button 616 launches the voicemail application 
and may begin playing recorded voicemail messages. 
Additionally, the voicemail application may present a list of 
voicemail messages onscreen and allow a user to access and 
respond to voicemail messages using transport controls such as 

15 play, previous, and next. Once playback has stopped, the 
Voicemail button indicator is reset as unlit and a user may 
respond by calling the sender using the telephony controls or 
by sending a recorded message via the voicemail application. 
It will be appreciate by those skilled in the art that 

20 the telephony controls 410 illustrated in FIG. 6 as 

incorporated in a keyboard is an exemplary configuration and 
that the present invention may be practiced using other 
configurations such as the configuration illustrated in FIG. 7 
with the telephony controls located on a communications panel. 

- 37 - 


In FIG. 7, two telephony controls 410, the Speakerphone button 
604 and the Voicemail button 614, are shown magnified from 
their location on a control panel 740 which is attached to or 
underneath the main display 191. This location may allow good 
5 visibility of the telephony controls 410, but it may not be 
considered as convenient by some user to press the telephony 
control buttons as it would be if they were located on the 
keyboard as illustrated in FIG. 6. FIG. 8 shows another 
exemplary illustration of the same two telephony controls 
10 located on the bezel of a monitor 191. Other telephony 
controls 410 may be located elsewhere. As discussed 
previously, the Talk button 602 is usually integrated into the 
keypad of telephone devices, such as a headset 306 or handset 
204. 

15 Another alternative control panel 920 incorporating a few 

telephony controls 410 and including an auxiliary display 220 
is shown in FIG. 9. Note that the telephony controls 410 in 
FIG. 9 are not grouped necessarily together and may be 
arranged among other control buttons. For example, the 

20 Microphone button 614 is not adjacent to the Voicemail button 
616 and the Speakerphone button 604. Thus, one or more 
telephony control 410 buttons need not be grouped with other 
telephony control buttons, but may be grouped with some other 
set of buttons, or may be an independent button. In fact, the 
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Microphone button 614 may be placed on a microphone 163, 
headset 306, handset 204, and so forth* One or more of the 
telephony controls 410 may also be an add-on to a computer 
system, such as by providing a button coupled to a USB or 
5 infrared port, or a as a Bluetooth™ device or other wired or 
wireless connection, A wired or wireless telephony control 
button, such as constructed to be held by a user, may also be 
provided either stand-alone or among buttons of a remote 
control . 

10 FIG. 10 represents a general overview of the logic 400 

that summarizes various states of operation. For example, as 
represented in FIG. 10, the hook switch operates in a number 
of ways, including using it to terminate a call, lifting it to 
make. a receiver active and pick up waiting line, as a flash 

15 button to switch lines or terminate a call if no lines are 

waiting. The hook state may be toggled between "off hook" and 
"on hook" by either lifting the handset from the cradle with a 
mechanically integrated hook switch, pressing the Talk button 
602, or pressing the Speaker button 604. 

20 FIG. 10 also represents a number of user interaction 

scenarios with respect to real-time communications, including 
handling an incoming call and making a call. To make a phone 
call, the user lifts the handset or presses the Talk button 
602, or presses the Speakerphone button 604, and then dials a 
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number. Alternatively, the user may select a contact from the 
display 220 or graphical user interface 440, or employ voice 
dialing. When initiating a call from the graphical user 
interface 440, such as within the context of a communications 
5 application such as Microsoft® Outlook, the user can initiate 
a phone call from any standard contact object supported by an 
API. The operating system takes the default receiver off hook 
and makes the connection using a preferred or specified 
telecommunications service . 

10 To answer a phone call, the user lifts the handset, 

presses the Speakerphone button 604, or presses the Talk 
button 602. Before answering, the user has the ability to 
view caller ID, and possibly other pertinent contact 
information displayed with the notification message (on the 

15 main display or auxiliary display) , allowing the user to pre- 
screen the call. Depending on preferences, the user can then 
choose to pick up the call or transfer it to voice mail, which 
may include transferring it to one of a plurality of custom or 
standard voice mail messages. While the caller is recording 

20 voice mail, the user can monitor the incoming recording and 
pick up the receiver, similar to a traditional answering 
machine . 

The examples below provide additional interaction detail 
for common telephony functions, including an incoming call 
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scenario, wherein the real-time communications controller 
sounds the ringer and blinks the Talk button indicator, while 
using caller ID information to display contact information. 
When the user presses the Talk button, a Microphone button, 
5 camera button, or picks up the handset, or presses the hook 
(talk) switch on the headset, the real-time communications 
controller 300 may light an indicator for the active receiver. 
The user receives the call, and can switch among devices, 
e.g., to switch to the speakerphone, the user presses the 

10 Speakerphone button 604; to switch to the handset, the user 
picks it up (activating the hook switch) ; or to switch to the 
headset, the user presses the hook (talk) switch on the 
headset. A cell phone or mobile computing device may be 
similarly used, even without actually switching the call to 

15 the cellular network (which may require some additional user 
action) . When the user switches active receivers, the system 
disables the previous receiver to prevent feedback, and 
updates indicators to reflect the active receiver. 

If the call supports video, the real-time communications 

20 controller blinks the camera indicator. If the user presses 
the blinking camera button, or automatic video is set by user 
preference to automatically stream video when video is 
available, the system opens the video monitor, lights the 
camera indicator, and starts streaming video. 
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To mute a component, the user can take an appropriate 
action. For example, to mute video and audio, the user 
presses the camera button, whereby the real-time 
communications controller blinks the audio and video 
5 indicators until the camera button is pressed again or the 
call is terminated. To mute video only, the user shuts the 
camera's privacy shutter, whereby the system un-lights the 
video indicators and closes the A/V monitor, but maintains the 
audio indicator. To mute speakerphone audio, the user may 

10 press the Speakerphone button 604. To hang up, the user 
presses the hook switch for the active receiver such as 
placing the handset 204 in the cradle if the handset is the 
active receiver or pressing the Speakerphone button 604 if the 
speakerphone is the active receiver. 

15 To place a call, the user presses the Talk button 602, 

picks up the handset 204, presses the Speakerphone button 604, 
or presses the hook switch on headset. A dialog box is 
displayed for selecting the person to call. The system may 
also audibly ask through the active receiver, such as when 

20 voice dialing is enabled, as to who the user would like to 
call. The resulting input is used to select a contact from 
the dialog box, and the user can query to search and can re- 
query to refine the search, if necessary. 
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When the user makes a selection or acknowledges a match, 
contact information for the selected recipient is displayed 
with a status dialog box while the call is connecting. When 
connected, the user can switch devices as described above, and 
5 also switch modes, e.g., audio to video, and so forth, with 
the indicators operating in the above-described manner. 

The user can also perform other communications-related 
tasks, including checking voice mail and call screening. For 
example, an indicator light may blink to indicate a voice 

10 mail, whereby the user can press the Voicemail button 616 to 
access voice mail. In one embodiment, this open the user's 
unified In Box. Call screening logic follows various rules, 
including showing caller ID information and contact summary 
data until, after a preset number of rings, the system stops 

15 the ringer and sends a voice mail message to an incoming 

caller. If the call screening option is enabled, the system 
echoes the incoming message to a system audio output device 
(usually the speaker) . If the user presses the hook switch of 
any receiver, the system discards the message and switches to 

20 real-time voice, otherwise, when the user hangs up or after a 
certain time limit, the system stops blinking the Talk button 
indicator and files a voice message. 

As can be seen from the foregoing detailed description, 
there is provided a system and method including an enhanced 
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user interface with telephony controls for a personal computer 
with integrated telephony capabilities. These telephony 
controls allow users to intuitively activate and interact with 
telecommunications applications such as handling phone calls 
5 and voice mail. The system and method is fairly comprehensive 
in handling communications, as well as flexible and extensible 
to handle many user scenarios and usage patterns. Additional 
telephony controls may be easily added. The system and method 
thus provide significant advantages and benefits needed in 

10 contemporary computing and communications. 

While the invention is susceptible to various 
modifications and alternative constructions, certain 
illustrated embodiments thereof are shown in the drawings and 
have been described above in detail. It should be understood, 

15 however, that there is no intention to limit the invention to 
the specific forms disclosed, but on the contrary, the 
intention is to cover all modifications, alternative 
constructions, and equivalents falling within the spirit and 
scope of the invention. 
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